feat: version-aware MockServer abstraction for client scenarios#321
Open
felixweinberger wants to merge 6 commits into
Open
feat: version-aware MockServer abstraction for client scenarios#321felixweinberger wants to merge 6 commits into
felixweinberger wants to merge 6 commits into
Conversation
MockServer encapsulates the lifecycle scaffold a client-conformance scenario presents to the client-under-test: - createServerStateful: 2025-x lifecycle. SDK Server + StreamableHTTPServerTransport (sessionless mode); the SDK handles the initialize handshake. - createServerStateless: 2026-x lifecycle (SEP-2575). Raw express app that validates _meta + MCP-Protocol-Version on every request, serves server/discover, routes other methods to the supplied handlers. createServerFor(specVersion) picks the implementation. ScenarioContext bundles specVersion and a bound createServer() for the runner to hand to each scenario. This is the client-conformance mirror of src/connection (PR #318). Nothing uses it yet; wiring follows in the next commit.
Scenario.start() becomes start(ctx: ScenarioContext). The runner builds the context from --spec-version (defaulting to LATEST_SPEC_VERSION) and passes it through; scenarios receive it as _ctx and otherwise behave identically. No scenario uses ctx.createServer() yet, so behaviour is unchanged: 231/231 tests pass. Test files use a testScenarioContext() helper. The runner already threads MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client process, so the fixture-side env wiring is unchanged.
…t scenarios ToolsCallScenario now goes through ctx.createServer() instead of an inline express + SDK Server build. Same handlers, same checks; the assertion now reads from srv.recorded so it works regardless of which lifecycle scaffold the runner picked. initialize, sse-retry, and elicitation-defaults are tagged removedIn: DRAFT (initialize/GET-SSE/SSE-embedded-elicitation are gone in the 2026 lifecycle; the MRTR sibling for elicitation-defaults is a follow-up). spec-version.test.ts: the 'draft is a superset of latest' invariant no longer holds once removedIn: DRAFT exists; the test now asserts that any scenario in latest-but-not-draft is explicitly removedIn.
The auth helper now takes ctx: ScenarioContext as its first argument and branches on ctx.specVersion inside the /mcp route: the stateful path (SDK Server + StreamableHTTPServerTransport) is unchanged; under the draft version a raw stateless handler validates _meta + the MCP-Protocol-Version header, serves server/discover, and routes the same tools/list and tools/call responses. The PRM endpoint, bearer-auth middleware, and request logger sit above the branch and are version-independent. All 25 call sites across the 12 auth scenario files pass ctx through; ServerLifecycle and the express.Application return type are unchanged so stop()/getChecks() are untouched. Deviation from the MockServer wrapper approach: keeping the helper's return type as express.Application avoids restructuring 25 call sites' ServerLifecycle handling in this PR. Folding the auth seam onto ctx.createServer() fully is a follow-up once the lifecycle ownership moves into MockServer.
…PROTOCOL_VERSION Adds a statelessRequest(serverUrl, method, params) helper that POSTs with _meta + MCP-Protocol-Version (the SEP-2575 lifecycle), shimming around the SDK Client not yet supporting stateless mode. The runRequestMetadataClient handler's meta constants are extracted to share with the helper. runBasicClient (initialize, tools_call, json-schema-ref-no-deref) now branches on MCP_CONFORMANCE_PROTOCOL_VERSION: for the draft version it uses statelessRequest to call tools/list then tools/call; for dated versions it keeps the SDK Client path. The runner already passes MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client, so no runner change is needed.
…or, capability derivation, recorded parity, specVersion threading) - MockServerOptions removed (capabilities/configure had zero callers); opts param dropped from createServerStateful/Stateless/For and ScenarioContext. - validateStatelessRequest extracted from mock-server/stateless and exported; both the stateless MockServer and auth/helpers/createServer.ts call it so _meta/header/version validation cannot drift. - isStatefulVersion exported from connection/select; mock-server/select uses it instead of duplicating the version set. - runner/client.ts: env MCP_CONFORMANCE_PROTOCOL_VERSION set unconditionally to the resolved version; runInteractiveMode now takes specVersion and the CLI passes it. - createServerStateful: capabilities derived from handler method prefixes; newServer() moved inside the try so a capability mismatch surfaces as JSON-RPC -32603 instead of an HTML 500. Recording moved to the express layer so unregistered methods are captured (parity with stateless). - readFinalSseMessage return type now declares error.data. - Tests added for the capability derivation and unregistered-method recording.
commit: |
felixweinberger
commented
May 28, 2026
| readonly source = { introducedIn: '2025-11-25' } as const; | ||
| readonly source = { | ||
| introducedIn: '2025-11-25', | ||
| removedIn: DRAFT_PROTOCOL_VERSION |
Collaborator
Author
There was a problem hiding this comment.
this will need a new sibling
felixweinberger
commented
May 28, 2026
| import type { Scenario, ConformanceCheck, ScenarioUrls } from '../../types'; | ||
| import type { CallToolRequest } from '../../spec-types/2025-06-18'; | ||
|
|
||
| function createMcpServer(checks: ConformanceCheck[]): Server { |
Collaborator
Author
There was a problem hiding this comment.
boilerplate server setup now handled by mock-server/
felixweinberger
commented
May 28, 2026
| description = 'Tests calling tools with various parameter types'; | ||
| private app: express.Application | null = null; | ||
| private httpServer: any = null; | ||
| private srv: MockServer | null = null; |
Collaborator
Author
There was a problem hiding this comment.
MockServer becomes the general case for creating test servers rather than writing an express app inline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #318. Mirrors the server-side
Connection/RunContextabstraction for client conformance: scenarios that act as a mock server now get a version-awarectx.createServer(handlers)instead of building the server inline.Motivation and Context
Client-conformance scenarios spin up a mock server for the client-under-test to connect to. Today every scenario builds that server inline (express + SDK
Server+StreamableHTTPServerTransport, or rawhttp.createServer), which hard-codes the 2025 lifecycle. A scenario that should run under--spec-version draftcan't unless its mock server speaks the stateless lifecycle, and there are currently three different inline patterns for doing that.What changes
src/mock-server/{index,stateful,stateless,select,testing}.ts—MockServer = {url, recorded, close}with stateful (SDK-backed) and stateless (rawhttp.createServer, validates_meta, servesserver/discover) impls.createServerFor(specVersion)picks.validateStatelessRequest()is exported for reuse.Scenario.start()→start(ctx: ScenarioContext)whereScenarioContext = {specVersion, createServer()}. Runner builds it from--spec-versionand passes the resolved version to the spawned client process viaMCP_CONFORMANCE_PROTOCOL_VERSION(now set unconditionally;runInteractiveModealso takes it).isStatefulVersion()exported fromconnection/select.tsand reused bymock-server/select.tsand the auth helper, so the version boundary is defined once.client/tools_call.tsmigrated toctx.createServer(); assertions read fromsrv.recorded.removedIn: DRAFTon 3 client scenarios —initialize,sse-retry,elicitation-defaults.client/auth/helpers/createServer.tsis version-aware viactx.specVersion: for stateless versions it calls the sharedvalidateStatelessRequest()and routestools/list/tools/calldirectly. The 25 auth call sites now forwardctx. Full restructuring of the auth helper ontoctx.createServer()is deferred (would be ~200 more mechanical lines around theServerLifecycle.start(app)pattern).everything-client.ts: readsMCP_CONFORMANCE_PROTOCOL_VERSION;runBasicClient(handlestools_call) picks SDKClientvs a rawstatelessRequest()helper accordingly. Also fixes a pre-existing registration mismatch ('tools-call'vs'tools_call') that meant the carry-forward client test had never actually run against the fixture.prompts/listdoesn't trip the SDK's capability assertion;recordedcaptured at the express layer so it includes unregistered methods (parity with stateless).How Has This Been Tested?
233/233 unit tests, typecheck/lint/build clean.
tools_callagainsteverything-client: 1/1 under both--spec-version 2025-11-25and--spec-version draft.client/auth/index.test.ts: 43/43.Full client suite against
everything-client:2025-11-2516/18 scenarios (250 checks passed, 6 failed),draft26/32 scenarios (288 passed, 13 failed). Every draft-side failure is fixture-side (the SDK-basedeverything-clientcannot complete stateless MCP exchanges) or pre-existing on the base branch, not caused by this change; details in the matrix below.Per-scenario matrix
Applicable under both — 15
¹ fixture-gap:
everything-client's auth handlers still use the SDK (stateful) client, so scenarios that need a completed MCP exchange after auth can't finish statelessly until the SDK has stateless support.2025-only (
removedIn: DRAFT) — 3² pre-existing: these run against dedicated example clients (
sse-retry-test.ts,elicitation-defaults-test.ts), noteverything-client; unrelated to this PR. The twoauth/2025-03-26-*backcompat scenarios and the 3 extension scenarios sit outside the version axis and are unchanged.draft-only (
introducedIn: DRAFT) — 17³ passes but emits 0 checks; pre-existing, worth a separate look.
⁴ pre-existing on the base branch (2026-native scenarios deliberately untouched here; the
everything-clientfixes from #319 landed on main after this branch's base and will arrive with the rebase).New scenarios needed for draft parity — 1
InputRequiredResult.inputRequestswhose elicitationrequestedSchemacarries SEP-1034defaultvalues applies those defaults in itsinputResponsessep-2322-client-request-statecovers the MRTR re-call loop but not SEP-1034 defaults;elicitation-sep1034-client-defaultsisremovedIn: DRAFT(
initializeis covered byrequest-metadata;sse-retryhas no sibling because resumable streams are removed entirely.)Breaking Changes
Scenario.start()→start(ctx: ScenarioContext). All in-tree scenarios are updated.Types of changes
Checklist
Additional context
Deferred:
request-metadata,http-*,mrtr-client,json-schema-ref-deref) ontoMockServer— same call as feat: version-aware Connection abstraction for server scenarios #318 (DRAFT-scenario coherence pass).ctx.createServer()— version-awareness is achieved via the sharedvalidateStatelessRequest(); the express/ServerLifecycleshape is kept for now.